AMFuse: Add–Multiply-Based Cross-Modal Fusion Network for Multi-Spectral Semantic Segmentation

Multi-spectral semantic segmentation has shown great advantages under poor illumination conditions, especially for remote scene understanding of autonomous vehicles, since the thermal image can provide complementary information RGB image. However, methods to fuse from and are still under-explored. In this paper, we propose a simple but effective module, add–multiply fusion (AMFuse) fusion, consisting two math operations—addition multiplication. The addition operation focuses on extracting cross-modal features, while multiplication concentrates common features. Moreover, attention module atrous spatial pyramid pooling (ASPP) modules also incorporated into our proposed AMFuse modules, enhance multi-scale context information. Finally, in UNet-style encoder–decoder framework, ResNet model is adopted as encoder. As decoder part, obtained hierarchically merged layer-by-layer restore feature map resolution segmentation. experiments RGBT multi-spectral salient object detection demonstrate effectiveness fusing